Deep learning neural networks can be interpreted as discretisations of an optimal control problem subject to an ordinary differential equation constraint. We discuss a number of interesting directions of research in structure preserving deep learning. Some deep neural networks can be designed to have desirable properties such as invertibility and group equivariance or can be adapted to problems of manifold value data. Equivariant neural networks are effective in reducing the amount of data for solving certain imaging problems. We show how classical results of stability of ODEs are useful to construct contractive neural networks architectures. We also consider applications of deep learning to mechanical systems, for learning Hamiltonians on manifolds and from noisy data and for learning PDE solutions.